{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro to Machine Learning with Python\n",
    "\n",
    "## Info\n",
    "- Scott Bailey (CIDR), *scottbailey@stanford.edu*\n",
    "- Javier de la Rosa (CIDR), *versae@stanford.edu*\n",
    "- Ashley Jester (CIDR/SSDS), *ajester@stanford.edu*\n",
    "\n",
    "## What are we covering today?\n",
    "- What is AI? What is ML?\n",
    "- Learning styles\n",
    "- Classification workflow with `scikit-learn`\n",
    "- Text classification\n",
    "- Training, testing and validation\n",
    "- Model evaluation and selection\n",
    "\n",
    "## Requirements\n",
    "Add `conda-forge` as a new channel in Anaconda.\n",
    "- `jupyter`\n",
    "- `scikit-learn`\n",
    "- `nltk`\n",
    "- `matplotlb`\n",
    "- `lime`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Goals\n",
    "\n",
    "By the end of the workshop, we hope you'll be able to identify when a problem related to a textual dataset might be approached using a classification strategy. Moreover, we hope you'll get a basic understanding of the different steps involved in the general workflow of machine learning using `scikit-learn`. We will be learning to use document classification to sort literary text by genre."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is AI? What is ML?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> In computer science, the field of **AI** research defines itself as the study of \"intelligent agents\": any device that perceives its environment and takes actions that maximize its chance of success at some goal.\n",
    "\n",
    "<div align=\"right\">&mdash; Source: [Wikipedia](https://en.wikipedia.org/wiki/Artificial_intelligence)</div>\n",
    "\n",
    "> Evolved from the study of pattern recognition and computational learning theory in artificial intelligence, **machine learning** explores the study and construction of algorithms that can learn from and make predictions on data – such algorithms overcome following strictly static program instructions by making data-driven predictions or decisions, through building a **model** from inputs.\n",
    "\n",
    "<div align=\"right\">&mdash; Source: [Wikipedia](https://en.wikipedia.org/wiki/Machine_learning)</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<figure>\n",
    "  <img style=\"margin: 0 0 -55px 0;\" src=\"https://media.licdn.com/mpr/mpr/AAEAAQAAAAAAAAhLAAAAJDFiODI3ZmNmLTFkOTgtNGYzZS04MWMzLTlkNmVmYjU2NTVlNw.png\" />  <figcaption><div align=\"right\" style=\"padding-top: 4px;\">&mdash; Source: <a href=\"https://www.linkedin.com/pulse/deep-learning-next-big-thing-adtech-volker-ballueder\">Is Deep Learning the next big thing in adtech?</a></div></figcaption>\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning styles\n",
    "\n",
    "Depending on the *input*: \n",
    "- Supervised: learning a mapping between example inputs and desired outputs.\n",
    "- Unsupervised: learning patterns and structure in unlabeled inputs.\n",
    "- Reinforcement: learning how to achieve a goal in a dynamic environment by giving feedback in terms of rewards and punishments.\n",
    "\n",
    "Depending on the *output*: \n",
    "- Supervised:\n",
    "  - Classification: inputs are labeled with one or more classes, and the learner, called the **classifier**, must produce a model that assigns unseen inputs to one or more of these classes. *Example*: Spam filtering is an example of classification, where the inputs are email messages and the classes are \"spam\" and \"not spam\".\n",
    "    \n",
    "    Algorithms:\n",
    "    - Support Vector Machines\n",
    "    - Decision Trees\n",
    "    - AdaBoost\n",
    "    - Gradient Boosting\n",
    "    - Random Forest\n",
    "    - Logistic Regression\n",
    "    - Neural Networks\n",
    "    - Maximum Entropy Classifier\n",
    "    - k-Nearest Neighbor\n",
    "    - Naïve Bayesian\n",
    "    - Discriminant Analysis\n",
    "\n",
    "  - Regression: inputs are continuous rather than discrete, and the learner is referred to as the **regressor**.\n",
    "    \n",
    "    Algorithms:\n",
    "    - Support Vector Regression\n",
    "    - Gaussian Process\n",
    "    - Regression Trees\n",
    "    - Gradient Boosting\n",
    "    - Random Forest\n",
    "    - RBF Networks\n",
    "    - OLS\n",
    "    - LASSO\n",
    "    - Ridge Regression\n",
    "\n",
    "- Unsupervised:\n",
    "  - Clustering: inputs' memberships are not known, and the learner must divide the inputs into groups, or **clusters**, by some kind of similarity measure.\n",
    "    \n",
    "    Algorithms:\n",
    "    - DBScan\n",
    "    - K-Means\n",
    "    - Hierarchical Clustering\n",
    "    - Self-Organizing Maps\n",
    "    - Spectral Clustering\n",
    "    - Minimum Entropy Clustering\n",
    "\n",
    "  - Dimensionality reduction: inputs' properties are reduced to a fewer number by keeping the most informative, **feature selection**, or by transforming them into a space of fewer dimensions, **feature extraction**. \n",
    "    \n",
    "    Algorithms:\n",
    "    - Principal Components Analysis\n",
    "    - Kernel PCA\n",
    "    - Linear Discriminant Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification workflow with `scikit-learn`\n",
    "\n",
    "Generally, the workflow for any classification task is usually as follows: \n",
    "1. **Collect** or create clean **labeled data**\n",
    "2. **Transform** that data into a numeric representation\n",
    "  - Each numeric value representing a characteristic of the data is called a **feature**\n",
    "  - The set of all features representing a single pair of input data and label is called the **feature vector**\n",
    "  - The whole data is split into at least a training set and a test set\n",
    "3. **Train**, learn, or fit a model on a part of the transformed labeled data (training set)\n",
    "4. **Test** the model predictions on unseen transformed labeled data (test set) to evaluate its performance\n",
    "5. Assess your model and tweak each of the previous steps\n",
    "\n",
    "### `scikit-learn`\n",
    "\n",
    "In Python, one solid choice for machine learning is the library [`scikit-learn`](http://scikit-learn.org/stable/):\n",
    "- Simple and efficient tools for data mining and data analysis\n",
    "- Accessible to everybody, and reusable in various contexts\n",
    "- Built on NumPy, SciPy, and matplotlib\n",
    "- Open source, commercially usable - BSD license\n",
    "\n",
    "### Simple and unified API\n",
    "All learning algorithms in `scikit-learn` share a uniform and limited API consisting of complementary interfaces:\n",
    "- An `estimator` interface for building and fitting models\n",
    "- A `predictor` interface for making predictions\n",
    "- A `transformer` interface for converting data\n",
    "\n",
    "The goal is to enforce a simple and consistent API to make it trivial to swap or plug algorithms.\n",
    "\n",
    "Estimators\n",
    "\n",
    "```python\n",
    "class Estimator(object):\n",
    "    def fit(self, X, y=None):\n",
    "        \"\"\"Fits estimator to data.\"\"\"\n",
    "        ...\n",
    "        return self\n",
    "    \n",
    "    def predict(self, X):\n",
    "        \"\"\"Predict class for unseen data.\"\"\"\n",
    "        ...\n",
    "        return\n",
    "    \n",
    "    def transform(self):\n",
    "        \"\"\"Converts data.\"\"\"\n",
    "        ...\n",
    "        return\n",
    "    \n",
    "    def fit_transform(self, X, y):\n",
    "        \"\"\"Fit and then transform data.\"\"\"\n",
    "        ...\n",
    "        return\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text classification\n",
    "\n",
    "\n",
    "In the specific task of assigning a category to a text:\n",
    "1. Collect. For example, in the case of tweets, it would require downloading the tweets using the Twitter API, extracting the text from the JSON response, and assigning a label, like a specific hashtag.\n",
    "2. Transform. There are many strategies for turning your textual data into numbers. One of the most used and reliable for text classification is to look at occurrences of words in documents (Bag of Words or BOW), or counting them (BoW counts), which in `scikit-learn` is a form of `Vectorizer` known as the `CountVectorizer`. One of the decisions to make in this type of transformation is the number of words we are going to count. This is a **hyper-parameter**.\n",
    "3. Train. After selecting a model for classification, we train on part of the input data.\n",
    "4. Test. We then test on the remainder.\n",
    "5. Asses the model performance and tweak the hyper-parameters if needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's import `scikit-learn`'s `CountVectorizer`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "documents = [\n",
    "    \"This is a piece of text. This is also some random text. Also text.\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "count_vectorizer = CountVectorizer()\n",
    "count_vectorizer.fit(documents)\n",
    "print(\"Vocabulary size:\", len(count_vectorizer.vocabulary_))\n",
    "count_vectorizer.vocabulary_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "counts = count_vectorizer.transform(documents)\n",
    "print(counts)\n",
    "print(\"   ^  ^         ^\\n   |  |         |\\n  doc word_id count\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "doc = 0  # first document\n",
    "for word, word_id in count_vectorizer.vocabulary_.items():\n",
    "    print(word, \":\", counts[doc, word_id])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "counts = count_vectorizer.fit_transform(documents)\n",
    "print(counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`CountVectorizer` also has some options to disregard stopwords, count ngrams instead of words, cap the max number of words to count, normalize spelling, or count terms within a frequency range. It is worth exploring the [documentation](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"font-size: 1em; margin: 1em 0 1em 0; border: 1px solid #86989B; background-color: #f7f7f7; padding: 0;\">\n",
    "<p style=\"margin: 0; padding: 0.1em 0 0.1em 0.5em; color: white; border-bottom: 1px solid #86989B; font-weight: bold; background-color: #AFC1C4;\">\n",
    "Activity\n",
    "</p>\n",
    "<p style=\"margin: 0.5em 1em 0.5em 1em; padding: 0;\">\n",
    "Fill in the text in the code cell below so after transforming it using a `CountVectorizer` the counts are as shown (order of words is not important).\n",
    "\n",
    "```\n",
    "flowers : 1\n",
    "garden : 1\n",
    "up : 1\n",
    "some : 1\n",
    "the : 1\n",
    "morning : 1\n",
    "place : 1\n",
    "every : 1\n",
    "pick : 1\n",
    "from : 1\n",
    "my : 1\n",
    "by : 1\n",
    "```\n",
    "<!--\n",
    "<br/>\n",
    "* **Hint**: You could count the characters in the words.*\n",
    "-->\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "documents = [\n",
    "    ...\n",
    "]\n",
    "count_vectorizer = CountVectorizer()\n",
    "counts = count_vectorizer.fit_transform(documents)\n",
    "doc = 0  # first document\n",
    "for word, word_id in count_vectorizer.vocabulary_.items():\n",
    "    print(word, \":\", counts[doc,word_id])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training, testing, and validation\n",
    "\n",
    "`scikit-learn` provides functions to split a dataset into training and testing. Often times, if part of the data is used to fine tune the model, holding out another split for validation is a good idea.\n",
    "\n",
    "For our purposes, we will be using the [Brown corpus](http://icame.uib.no/brown/bcm-los.html) included in NLTK, which comprises more than a million words of English text, containing text from 500 sources, and categorized by genre. Specifically, we will use 2 categories: `fiction` (W.E.B. Du Bois: *Worlds of Color*), and `romance` (Callaghan: *A Passion in Rome*). The goal will be to create a classifier able to assign a text to `romance` or `fiction` only based on its textual data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "nltk.download('all', quiet=True);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import random\n",
    "from nltk.corpus import brown\n",
    "\n",
    "dataset = []\n",
    "categories = ['fiction', 'romance']\n",
    "for category in categories:\n",
    "    for fileid in brown.fileids(category):\n",
    "        text = \" \".join(brown.words(fileids=fileid))\n",
    "        dataset.append((text, category))\n",
    "\n",
    "random.shuffle(dataset)\n",
    "print(len(dataset), \"documents:\", \", \".join(\" \".join((str(len(brown.fileids(c))), c)) for c in categories))\n",
    "\n",
    "\"...\" + text[35:166] + \"...\", category"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the dataset, we can now separate the labels from the texts into two different variables. Usually, the varaible containing the labels is named `y`, and the one containing the array of features (in our case, the transformed version of the texts) is named `X`, as in you can obtain the output `y` as a function of the inputs `X`, which is the core abstraction in `scikit-learn`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np  # scikit-learn works internally with NumPy arrays\n",
    "\n",
    "texts = []\n",
    "labels = []\n",
    "for text, label in dataset:\n",
    "    texts.append(text)\n",
    "    labels.append(label)\n",
    "    \n",
    "X = np.array(texts)\n",
    "y = np.array(labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "(X_train_texts, X_test_texts,\n",
    " y_train, y_test) = train_test_split(X, y, test_size=0.25, random_state=42)\n",
    "\n",
    "print(\"{} training documents\".format(*X_train_texts.shape))\n",
    "print(\"{} testing documents\".format(*X_test_texts.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Don't forget to transform the text!\n",
    "vectorizer = CountVectorizer()\n",
    "X_train = vectorizer.fit_transform(X_train_texts)\n",
    "X_test = vectorizer.transform(X_test_texts)\n",
    "\n",
    "print(\"{} training documents with {} features per document\".format(*X_train.shape))\n",
    "print(\"{} testing documents with {} features per document\".format(*X_test.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with one the Naïve Bayes classifiers.\n",
    "\n",
    "**Naïve Bayes** are a family of classifiers based on the Bayes Theorem for probability, which describes the probability of an event based on its prior knowledge. Although its formulation can get confusing, all the math boils down to counting, multiplication and division, making Naïve Bayes classifiers very fast. On the other hand, it makes the assumption that all of the features in the data set are equally important and independent, which is rarely true in real life.\n",
    "\n",
    "Based on its characteristics, Naïve Bayes classifiers are well suited to perform text classification, where word counts are considered to be independent features.\n",
    "\n",
    "There are three Naïve Bayes algorithms in `scikit-learn`: \n",
    "- Gaussian. It assumes that features follow a normal distribution.\n",
    "- Multinomial. Good for discrete counts, like in text classification problems using counts of words.\n",
    "- Bernoulli. Useful for feature vectors that are binary (i.e. zeros and ones), like classic bag of words.\n",
    "\n",
    "Given that our feature vectors are counts of bag of words, we will use `MultinomialNB`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "\n",
    "clf = MultinomialNB()\n",
    "\n",
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can predict the categories of the held out texts and assess how well our classifier did."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "samples = [\n",
    "    \"She will travel at the speed of light in that rocket\",\n",
    "]\n",
    "transformed_samples = vectorizer.transform(samples)\n",
    "clf.predict(transformed_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we wanted to know which words are being used to decide whether a sentence is either `romance` or `fiction`, we need to take into account the coefficients calculated by the model at training time, and the vocabulary built by the  vectorizer we used to transform our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def most_informative_features(classifier, vectorizer=None, n=10):\n",
    "    class_labels = classifier.classes_\n",
    "    if vectorizer is None:\n",
    "        feature_names = classifier.steps[0].get_feature_names()\n",
    "    else:\n",
    "        feature_names = vectorizer.get_feature_names()\n",
    "    topn_class1 = sorted(zip(classifier.coef_[0], feature_names))[:n]\n",
    "    topn_class2 = sorted(zip(classifier.coef_[0], feature_names))[-n:]\n",
    "    for coef, feat in reversed(topn_class2):\n",
    "        print(class_labels[1], coef, feat)\n",
    "    print()\n",
    "    for coef, feat in topn_class1:\n",
    "        print(class_labels[0], coef, feat)\n",
    "\n",
    "most_informative_features(clf, vectorizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%capture --no-display\n",
    "# Do not worry about this particular chunk of code\n",
    "\n",
    "%matplotlib inline\n",
    "from lime.lime_text import LimeTextExplainer\n",
    "from sklearn.pipeline import make_pipeline\n",
    "\n",
    "def explain(entry, clf, vectorizer=None, n=10):\n",
    "    if vectorizer is None:\n",
    "        class_names = clf.steps[1].classes_.tolist()\n",
    "        pipeline = clf\n",
    "    else:\n",
    "        class_names = clf.classes_.tolist()\n",
    "        pipeline = make_pipeline(vectorizer, clf)\n",
    "    explainer = LimeTextExplainer(class_names=class_names)\n",
    "    exp = explainer.explain_instance(entry, pipeline.predict_proba, num_features=n)\n",
    "    exp.show_in_notebook()\n",
    "\n",
    "explain(samples[0], clf, vectorizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That doesn't seem very good. How can we better assess and improve the performance of our model?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"font-size: 1em; margin: 1em 0 1em 0; border: 1px solid #86989B; background-color: #f7f7f7; padding: 0;\">\n",
    "<p style=\"margin: 0; padding: 0.1em 0 0.1em 0.5em; color: white; border-bottom: 1px solid #86989B; font-weight: bold; background-color: #AFC1C4;\">\n",
    "Activity\n",
    "</p>\n",
    "<p style=\"margin: 0.5em 1em 0.5em 1em; padding: 0;\">\n",
    "[`MultinomialNB`](http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html) has some hyper-parameters that can be fine-tuned. Play around with the `alpha` (from `0` to `1`) and `fit_prior` (`True` or `False`) to see if you can get \"better\" important features.\n",
    "<!--\n",
    "<br/>\n",
    "* **Hint**: Y.*\n",
    "-->\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "alpha = 0.1\n",
    "fit_prior = False\n",
    "\n",
    "most_informative_features(\n",
    "    MultinomialNB(alpha=alpha, fit_prior=fit_prior).fit(X_train, y_train),\n",
    "    vectorizer\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model evaluation and selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Changing a model's hyper-parameters without really having a way to assess its performance can get problematic. If, for example, you find better informative features but the classifier is failing 50% of the time, it doesn't really matter much how good you think those features are. In some cases, very performant classifiers make use of unexpected or counterintuitive features.\n",
    "\n",
    "The only real way to assess its performance is by making the classifier predict labels for unseen data, and then comparing the predicted labels with the real labels. This is where the importance of separating training and testing data lies in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_pred = clf.predict(X_test)\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"Label     Predicted   Result\\n-----     ---------   ------\")\n",
    "for i, real_label in enumerate(y_test):\n",
    "    predicted_label = y_pred[i]\n",
    "    if real_label == predicted_label:\n",
    "        result =  \"hit\"\n",
    "    else:\n",
    "        result = \"miss\"\n",
    "    print(real_label, \" \", predicted_label, \"   \", result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While hit vs. miss gives an idea of what's going on, it doesn't tell us much about the nature of the fails. Were the `romance` texts being classified as `fiction`, or the other way around? This might not seem very important, but if instead of texts you were trying to test whether a person has a deadly disease or not (here labels are positive, has the disease, or negative, does not have the disease), you might be more interested in classifiers with lower false negative ratio.\n",
    "\n",
    "Taking this into account, we could count how many `romance` texts have been correctly classified as `romance` (we will call this `TR`, as in true romance), and how many have been incorrectly classified as `fiction` (`FF`, as in false fiction); and the same for `fiction`: how many `fiction` texts have been classified as `fiction` (`TF`, as in true fiction), and how many as `romance` (`FR`, as in false romance)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"Label     Predicted   Result\\n-----     ---------   ------\")\n",
    "counts = {\"TR\": 0, \"TF\": 0, \"FF\": 0, \"FR\": 0}\n",
    "for i, real_label in enumerate(y_test):\n",
    "    predicted_label = y_pred[i]\n",
    "    if real_label == predicted_label:\n",
    "        if real_label == \"romance\":\n",
    "            result =  \"TR\"\n",
    "        else:\n",
    "            result = \"TF\"\n",
    "    else:\n",
    "        if real_label == \"romance\":\n",
    "            result =  \"FF\"\n",
    "        else:\n",
    "            result = \"FR\"\n",
    "    counts[result] += 1\n",
    "    print(real_label, \" \", predicted_label, \"   \", result)\n",
    "counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\" TF FR\")\n",
    "print([counts[\"TF\"], counts[\"FR\"]])\n",
    "print([counts[\"FF\"], counts[\"TR\"]])\n",
    "print(\" FF TR\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fortunately, `scikit-learn` comes with a function that calculates just that table, which is called the confusion matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "print(\" TF FR\")\n",
    "print(confusion_matrix(y_test, y_pred))\n",
    "print(\" FF TR\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "(TF, FR), (FF, TR) = confusion_matrix(y_test, y_pred)\n",
    "print(\" TF FR\")\n",
    "print([TF, FR])\n",
    "print([FF, TR])\n",
    "print(\" FF TR\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Depending on your problem, inspecting the confusion matrix might not be necessary, as in the case of distinguishing a `romance` from a `fiction` text, as long as the overall performance is good. **Accuracy** is one of those measures that gives an overall value of performance. If the accuracy is 0, that means the classifier is not doing a good job; if 0.5, that means it's failing 50% of the time, basically as good as a random classifier; if 1 the predictions are perfect."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"font-size: 1em; margin: 1em 0 1em 0; border: 1px solid #86989B; background-color: #f7f7f7; padding: 0;\">\n",
    "<p style=\"margin: 0; padding: 0.1em 0 0.1em 0.5em; color: white; border-bottom: 1px solid #86989B; font-weight: bold; background-color: #AFC1C4;\">\n",
    "Activity\n",
    "</p>\n",
    "<p style=\"margin: 0.5em 1em 0.5em 1em; padding: 0;\">\n",
    "Accuracy is defined as the ratio of hits vs the total number of predictions. Using the numbers in the confusion matrix above, do the math to get the accuracy of our classifier.\n",
    "<br/>\n",
    "* **Hint**: Use the variables TR, TF, FR, and FF *\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "accuracy = ... / ...\n",
    "accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"{:.4}%\".format(accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or you can do it the `scikit-learn` way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "accuracy_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "alpha = 0.0000001\n",
    "fit_prior = False\n",
    "tuned_clf = MultinomialNB(alpha=alpha, fit_prior=fit_prior).fit(X_train, y_train)\n",
    "tuned_y_pred = tuned_clf.predict(X_test)\n",
    "accuracy_score(y_test, tuned_y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What are other moving parts involved in our workflow? Let's look at our vectorizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_accuracy(X, y, vectorizer):\n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)\n",
    "\n",
    "    X_train = vectorizer.fit_transform(X_train)\n",
    "    X_test = vectorizer.transform(X_test)\n",
    "\n",
    "    clf = MultinomialNB()\n",
    "    clf.fit(X_train, y_train)\n",
    "\n",
    "    y_pred = clf.predict(X_test)\n",
    "\n",
    "    return accuracy_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"font-size: 1em; margin: 1em 0 1em 0; border: 1px solid #86989B; background-color: #f7f7f7; padding: 0;\">\n",
    "<p style=\"margin: 0; padding: 0.1em 0 0.1em 0.5em; color: white; border-bottom: 1px solid #86989B; font-weight: bold; background-color: #AFC1C4;\">\n",
    "Activity\n",
    "</p>\n",
    "<p style=\"margin: 0.5em 1em 0.5em 1em; padding: 0;\">\n",
    "The function `get_accuracy()` defined below, takes in input data in `X` and its labels in `y`, and returns the accuracy of a `MultinomialNB` using `vectorizer` passed in as another argument. Using the function `get_accuracy()`, complete the code cell below so it performs a search over vocabulary size (10000, 1000, 500, 300, 150, 100) using and ignoring stop words (prepositions, articles, etc.). What set of parameters is performing better?\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "for stopwords in [None, 'english']:\n",
    "    for vocabulary_size in ...:\n",
    "        print(\"stopwords:\", ...)\n",
    "        print(\"vocabulary_size:\", ...)\n",
    "        vectorizer = CountVectorizer(...)\n",
    "        accuracy = ...\n",
    "        print(\"accuracy:\", accuracy)\n",
    "        print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, the split might also have an effect in the model performance. One common strategy to prevent the split from affecting our model, is to split the dataset randomly several times in different **folds**, calculate whatever measure you want, and then average the scores. This process is called **cross-validation**, and also prevents us from making the model learn too well the training data (**over-fitting**).\n",
    "\n",
    "In order to cross-validate our model, we need to create a pipeline in `scikit-learn` by combining both the vectorizer and the classifier, then calling the `cross_val_score` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "for stopwords in [None, 'english']:\n",
    "    for vocabulary_size in [10000, 1000, 500, 300, 150, 100]:\n",
    "        print(\"stopwords:\", stopwords)\n",
    "        print(\"vocabulary_size:\", vocabulary_size)\n",
    "        vectorizer = CountVectorizer(\n",
    "            max_features=vocabulary_size, stop_words=stopwords)\n",
    "        \n",
    "        pipe = make_pipeline(vectorizer, MultinomialNB())\n",
    "        accuracies = cross_val_score(pipe, X, y, cv=10)\n",
    "        \n",
    "        print(\"accuracy:\", accuracies.mean())\n",
    "        print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By looking at this data, which is more reliable, it looks like we could get a classifier with up to ~75% of accuracy. However, if we want a model more intuitive and with a better explanatory power, we could go with the model that uses stop words, although its accuracy is only ~65%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "vectorizer = CountVectorizer(max_features=100, stop_words=\"english\")\n",
    "X_train = vectorizer.fit_transform(X_train_texts)\n",
    "X_test = vectorizer.transform(X_test_texts)\n",
    "\n",
    "clf = MultinomialNB()\n",
    "clf.fit(X_train, y_train)\n",
    "y_pred = clf.predict(X_test)\n",
    "\n",
    "most_informative_features(clf, vectorizer, n=10)\n",
    "entry = -1\n",
    "print()\n",
    "print(\"Label:\", y_test[entry])\n",
    "print(\"Predicted:\", y_pred[entry])\n",
    "explain(X_test_texts[entry], clf, vectorizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"font-size: 1em; margin: 1em 0 1em 0; border: 1px solid #86989B; background-color: #f7f7f7; padding: 0;\">\n",
    "<p style=\"margin: 0; padding: 0.1em 0 0.1em 0.5em; color: white; border-bottom: 1px solid #86989B; font-weight: bold; background-color: #AFC1C4;\">\n",
    "Activity\n",
    "</p>\n",
    "<p style=\"margin: 0.5em 1em 0.5em 1em; padding: 0;\">\n",
    "In the code cell below, add a new for loop to the parameters search using cross-validation so other models (`LinearSVC`, `SVC`, and `RandomForestClassifier` with default parameters) are also used. What's the best model and parameters for our dataset?\n",
    "</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.svm import SVC, LinearSVC\n",
    "\n",
    "for ...:\n",
    "    for stopwords in [None, 'english']:\n",
    "        for vocabulary_size in [10000, 1000, 500, 300, 150, 100]:\n",
    "            print(\"model:\", ...)\n",
    "            print(\"stopwords:\", stopwords)\n",
    "            print(\"vocabulary_size:\", vocabulary_size)\n",
    "            vectorizer = CountVectorizer(\n",
    "                max_features=vocabulary_size, stop_words=stopwords)\n",
    "\n",
    "            pipe = make_pipeline(vectorizer, ...)\n",
    "            accuracies = cross_val_score(pipe, X, y, cv=10)\n",
    "\n",
    "            print(\"accuracy:\", accuracies.mean())\n",
    "            print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<figure>\n",
    "  <img  src=\"http://scikit-learn.org/stable/_static/ml_map.png\" />  <figcaption><div align=\"right\" style=\"padding-top: 4px;\">&mdash; Source: <a href=\"http://scikit-learn.org/stable/tutorial/machine_learning_map/\">`scikit-learn`: Choosing the right estimator</a></div></figcaption>\n",
    "</figure>"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
